======================
**AWS S3兼容文档**
======================


前言
============

介绍
^^^^^^^^^^^^^^^^^

  本文档主要是针对NOS兼容AWS S3协议后，与S3的一些异同


支持的SDK
===============
- Java SDK
- GO SDK
- Python SDK
- PHP SDK


支持的接口
==============

**Bucket**

* List Buckets
* Put Bucket
* Put Bucket Acl
* Get Bucket Acl
* Delete Bucket
* Get Bucket Location


**Object**

* Put Object
* Put Object Copy
* Get Object
* Delete Object
* Delete Multiple Objects

* Head Object
* List Objects

**Multipart Upload**

* Initiate Multipart Upload
* Upload Part
* List Parts
* List Multipart Uploads
* Complete Multipart Upload
* Abort Multipart Upload

约束说明
============

================= ==============
  参数              详细说明
================= ==============
 加密               暂不支持
 Acl               支持public-read,private   
 Grant             不支持
 Lifecycle         不支持 
 多版本             不支持 
 Region            支持us-east-1对应的分区为杭州
 StorageClass      默认为STANDARD,其他的存储级别暂不支持 
 Authorization     支持v4,v2，目前只支持在header加入签名信息，暂不支持url签名
 MFA授权            不支持
================= ==============

Bucket接口
====================


List Buckets
^^^^^^^^^^^^^^^^^^^
兼容说明
'''''''''
 接口和S3兼容

代码示例
'''''''''
Java::

 AWSCredentials credentials= new BasicAWSCredentials("ak","sk");
 AmazonS3 s3Client = new AmazonS3Client(credentials);
 s3Client.setEndpoint("http://xxx.com");//指定为nos的endpoint

 List<Bucket> listBuckets = s3Client.listBuckets();
 for(Bucket bucket : listBuckets){
     System.out.println(bucket.getOwner() + "-->" + bucket.getName() + "-->" +  bucket.getCreationDate());
 }

Put Bucket
^^^^^^^^^^^
兼容说明
'''''''''

 注意约束说明中的说明即可

代码示例
'''''''''
Java::

 s3Client.createBucket("testbucket");


Put Bucket Acl
^^^^^^^^^^^^^^^

兼容说明
'''''''''

 注意约束说明中的说明即可

代码示例
'''''''''
Java::

 s3Client.setBucketAcl(bucketName,CannedAccessControlList.PublicRead);

Get Bucket Acl
^^^^^^^^^^^^^^^^^^^^
兼容说明
'''''''''

- 因为NOS的权限不能赋给其他的用户，所以返回的Body中只有一条记录
- 在返回值中，Permission的取值有两种类型，Read(public-read)，Write(private)

代码示例
'''''''''
Java::

        AccessControlList accessControlList = s3Client.getBucketAcl(bucketName);
        System.out.println("owner : " +  accessControlList.getOwner().getId() + " : " + accessControlList.getOwner().getDisplayName());
        for(Grant grant : accessControlList.getGrantsAsList()){//由于不支持授权给其他用户，所以List中只存在一个Grant
            System.out.println(grant.getGrantee().getIdentifier() + " : " +  grant.getPermission() + " : " + grant.getGrantee().getTypeIdentifier());
        }

Delete Bucket
^^^^^^^^^^^^^^
兼容说明
'''''''''

  接口和S3兼容

代码示例
'''''''''
Java::

 s3Client.deleteBucket(testBucketName);

Get Bucket Location
^^^^^^^^^^^^^^^^^^^^

兼容说明
'''''''''

 接口和S3兼容

代码示例
'''''''''
Java::

 System.out.println(s3Client.getBucketLocation(testBucketName));


Object接口
====================

Put Object
^^^^^^^^^^^
兼容说明
'''''''''

- x-amz-tagging                     不支持
- x-amz-website-redirect-location   不支持
- Expires                           不支持
- Content-Encoding                  不支持
- Java SDK不支持http上传

代码示例
'''''''''
Java::

 String content = "test put object content";
 String testKey = "testKey";
 PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName,testKey,"/");
 ObjectMetadata objectMetadata = new ObjectMetadata();
 objectMetadata.setContentLength(content.length());
 Map<String,String> userMeta = new TreeMap<>();
 userMeta.put("userMeta1","meta1");
 userMeta.put("userMeta2","meta2");
 objectMetadata.setUserMetadata(userMeta);
 putObjectRequest.setMetadata(objectMetadata);
 putObjectRequest.setInputStream(new ByteInputStream(content.getBytes(Charset.forName("UTF-8")),content.getBytes(Charset.forName("UTF-8")).length));
 s3Client.putObject(putObjectRequest);


Put Object Copy
^^^^^^^^^^^^^^^^^

兼容说明
'''''''''

- x-amz-metadata-directive,不支持设置，用户原信息会复制过去
- x-amz-copy-xxx 等参数不支持
- x-amz-storage-class，存储级别使用默认(STANDARD)的就好，NOS不支持其他的存储级别
- x-amz-tagging-directive,x-amz-website-redirect-location 也不支持

代码示例
'''''''''
Java::

  CopyObjectResult copyObjectResult = s3Client.copyObject(srcBucket,srcKey,destBucket,destKey);
  System.out.println(copyObjectResult.getETag());
  System.out.println(copyObjectResult.getLastModifiedDate());

Get Object
^^^^^^^^^^^^^
兼容说明
'''''''''

- NOS不支持的操作，相关的响应头是不会返回的
- If-Unmodified-Since，If-Match,If-None-Match 不支持

代码示例
'''''''''
Java::


 String testKey = "testKey";
 GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName,testKey);
 S3Object s3Object = s3Client.getObject(getObjectRequest);
 System.out.println(s3Object.getObjectMetadata().getContentLength());
 if(s3Object != null){
    S3ObjectInputStream s3ObjectInputStream = s3Object.getObjectContent();
    byte[] buf = new byte[1024];
    int readLen;
    try {
        while((readLen = s3ObjectInputStream.read(buf)) > -1){
            System.out.println(new String(buf,0,readLen));
        }
        System.out.println(s3Object.getObjectMetadata().getUserMetadata());
    } catch (IOException e) {
        e.printStackTrace();
    }
 }          


Delete Object
^^^^^^^^^^^^^^^

兼容说明
'''''''''

- NOS会直接删除对象
- 删除对象只需要正确的ak/sk
- 返回值中不会包含x-amz-delete-marker和x-amz-version-id

代码示例
'''''''''
Java::

 s3Client.deleteObject(bucketName,testKey);

Delete Multiple Objects
^^^^^^^^^^^^^^^^^^^^^^^^^

兼容说明
'''''''''

 接口和S3兼容

代码示例
'''''''''
Java::

 String testBucketName  = bucketName;
 String testKey = "test";
 for(int i = 0;i < 5;++i){
     s3Client.putObject(testBucketName,testKey + i,"test" + i);
 }
 DeleteObjectsRequest deleteObjectsRequest = new DeleteObjectsRequest(testBucketName);
 List<DeleteObjectsRequest.KeyVersion> list = new LinkedList<>();
 for(int i = 0; i < 3;++i){
     list.add(new DeleteObjectsRequest.KeyVersion(testKey + i));
 }
 deleteObjectsRequest.setKeys(list);
 deleteObjectsRequest.setQuiet(true);
 DeleteObjectsResult deleteObjectsResult = s3Client.deleteObjects(deleteObjectsRequest);
 for(DeleteObjectsResult.DeletedObject deleteObject : deleteObjectsResult.getDeletedObjects()){
     System.out.println(deleteObject.getKey());
 }

	
Head Object
^^^^^^^^^^^^^

兼容说明
'''''''''

- 和GetObject一样，也不支持If-Unmodified-Since，If-Match,If-None-Match参数
- 同样的，加密也不支持
- 返回中带有Object该有的信息

代码示例
'''''''''
Java::

 ObjectMetadata objectMetadata =  s3Client.getObjectMetadata(bucketName,testKey);
 System.out.println(objectMetadata.getUserMetadata());
 System.out.println(objectMetadata.getContentLength());
 System.out.println(objectMetadata.getETag());

List Objects
^^^^^^^^^^^^^^^

兼容说明
'''''''''

 S3的List Objects有两个版本(v1,v2),NOS目前只兼容v1

代码示例
'''''''''
Java::

 ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
 listObjectsRequest.setBucketName("sdktest-private");
 List<S3ObjectSummary> listResult = new ArrayList<>();
 ObjectListing listObjects = s3Client.listObjects(listObjectsRequest);
 do {
     listResult.addAll(listObjects.getObjectSummaries());
     if (listObjects.isTruncated()) {
         ListObjectsRequest request = new ListObjectsRequest();
         request.setBucketName(listObjectsRequest.getBucketName());
         request.setMarker(listObjects.getNextMarker());
         listObjects =  s3Client.listObjects(request);
     } else {
         break;
     }
 } while (listObjects != null);



Multipart Upload接口
========================

Initiate Multipart Upload
^^^^^^^^^^^^^^^^^^^^^^^^^^

兼容说明
'''''''''

 x-amz-website-redirect-location，不支持

代码示例
'''''''''
Java::

 InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(
                bucketName, key);
 InitiateMultipartUploadResult initResponse =
                s3Client.initiateMultipartUpload(initRequest);


Upload Part
^^^^^^^^^^^^

兼容说明
'''''''''

 接口和S3兼容

代码示例
'''''''''
Java::

 UploadPartRequest uploadRequest = new UploadPartRequest()
                        .withBucketName(bucketName).withKey(key)
                        .withUploadId(initResponse.getUploadId()).withPartNumber(i)
                        .withFileOffset(filePosition)
                        .withFile(file)
                        .withPartSize(partSize);

 s3Client.uploadPart(uploadRequest)

List Parts
^^^^^^^^^^^^

兼容说明
'''''''''

- 对于s3中的encoding头，如果包含了，也不处理
- 因为nos没有生命周期的概念，所以响应中不会加入lifecycle相关的响应头

代码示例
'''''''''
Java::

 ListPartsRequest listPartsRequest = new ListPartsRequest(bucketName,key,initResponse.getUploadId());
 //initResponse通过上面的init方式获取
 s3Client.listParts(listPartsRequest);


List Multipart Uploads
^^^^^^^^^^^^^^^^^^^^^^^

兼容说明
'''''''''

- request只支持max-uploads,key-marker参数
- response中，因为不支持有的请求参数，所以有的元素也不会返回,详细见下表：

+-----------------------------------+------------+-----------+ 
|resp header                        |     S3     |    NOS    | 
+===================================+============+===========+ 
| UploadIdMarker                    | √          | x         | 
+-----------------------------------+------------+-----------+ 
| NextUploadIdMarker                | √          | x         | 
+-----------------------------------+------------+-----------+ 
| Encoding-Type                     | √          | x         | 
+-----------------------------------+------------+-----------+  
| Initiator                         | √          | x         | 
+-----------------------------------+------------+-----------+ 
| CommonPrefixes                    | √          | x         |
+-----------------------------------+------------+-----------+ 
| CommonPrefixes.Prefix             | √          | x         | 
+-----------------------------------+------------+-----------+ 


代码示例
'''''''''
Java::

 ListMultipartUploadsRequest listMultipartUploadsRequest = new ListMultipartUploadsRequest(bucketName);
 listMultipartUploadsRequest.setMaxUploads(2);
 MultipartUploadListing multipartUploadListing  = s3Client.listMultipartUploads(listMultipartUploadsRequest);
 System.out.println("bucketName : " + multipartUploadListing.getBucketName());
 System.out.println(multipartUploadListing.getMaxUploads());
 for(MultipartUpload multipartUpload : multipartUploadListing.getMultipartUploads()){
      System.out.println(multipartUpload.getInitiated() + " : " + multipartUpload.getUploadId() + " : " +
                    multipartUpload.getStorageClass());
 }



Complete Multipart Upload
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

兼容说明
'''''''''

 除了版本控制和加密，NOS完全兼容S3

代码示例
'''''''''
Java::

 CompleteMultipartUploadRequest compRequest = new
        CompleteMultipartUploadRequest(bucketName,
        key,
        initResponse.getUploadId(),//为initiateMultipartUpload的返回值
        partETags); //partEtags通过上述ListParts构造


Abort Multipart Upload
^^^^^^^^^^^^^^^^^^^^^^^^^

兼容说明
'''''''''

 接口和S3兼容

代码示例
'''''''''
Java::

 s3Client.abortMultipartUpload(new AbortMultipartUploadRequest(
                    bucketName, key, initResponse.getUploadId()));//initResponse为initiateMultipartUpload的返回值

完整的分块上传示例
^^^^^^^^^^^^^^^^^^

Java::

 List<PartETag> partETags = new ArrayList<>();
 // Step 1: Initialize.
 InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(
        bucketName, key);
 InitiateMultipartUploadResult initResponse =
        s3Client.initiateMultipartUpload(initRequest);
 File file = new File(filePath);
 long contentLength = file.length();
 long partSize = 5 * 1024 * 1024; // Set part size to 5 MB.
 try {
    // Step 2: Upload parts.
    long filePosition = 0;
    for (int i = 1; filePosition < contentLength; i++) {
        // Last part can be less than 5 MB. Adjust part size.
        partSize = Math.min(partSize, (contentLength - filePosition));

        // Create request to upload a part.
        UploadPartRequest uploadRequest = new UploadPartRequest()
                .withBucketName(bucketName).withKey(key)
                .withUploadId(initResponse.getUploadId()).withPartNumber(i)
                .withFileOffset(filePosition)
                .withFile(file)
                .withPartSize(partSize);
        // Upload part and add response to our list.
        partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());
        ListPartsRequest listPartsRequest = new ListPartsRequest(bucketName,key,initResponse.getUploadId());
        s3Client.listParts(listPartsRequest);

        filePosition += partSize;
    }
    // Step 3: Complete.
    CompleteMultipartUploadRequest compRequest = new
            CompleteMultipartUploadRequest(bucketName,
            key,
            initResponse.getUploadId(),
            partETags);
    s3Client.completeMultipartUpload(compRequest);
 } catch (Exception e) {
    //error,abort
    s3Client.abortMultipartUpload(new AbortMultipartUploadRequest(
            bucketName, key, initResponse.getUploadId()));
 }